A problem that continues to plague the land mobile radio (LMR) industry, its markets, and its end-users, is the misinterpretation of data resulting from coverage acceptance testing. The earliest technique used was a “voice test” where radio calls were made to and from selected locations in the service area. A standardized circuit merit (CM) scale of values ranging from 0 (unusable signal) to 5 (clear audio, speech easily understood) was used to score the test. Performance of the radio in both the talk-out (base-to-mobile) and talk-back (mobile-to-base) paths could simply be tested with a pair of test calls. If the persons performing the test were unable to reach a consensus about the score for the test, a mean opinion scoring (MOS) method could be applied where two or more individuals independently evaluate each call and assign a score. The final score ascribed to the test point would then be the average (or mean) score for the team. Since tests conducted at a stationary location do not include fading effects, test procedures were sometimes written to require that tests be performed in a moving vehicle.
Since circuit merit scoring involves subjective opinions concerning the audio quality of radio calls, an objective measurement method evolved utilizing signal strength as the pass/fail metric. In this method, the relationship between signal strength and delivered audio quality (DAQ) was assumed based generally on experience, and an appropriate threshold value of signal level was agreed upon as the boundary between acceptable and unacceptable performance. The establishment of a threshold signal level made certain assumptions about received signal strength and circuit merit that were generally true, but did not take into account individual receiver sensitivity or performance, multi-path effects, actual local noise, and others. The correlation was then “generally” good for comparable products manufactured by the major LMR equipment suppliers. The correlation held for analog narrow-band frequency modulation (FM) technology, but became less accurate when other forms of modulation were introduced.
It quickly became apparent that changes in signal strength due to multi-path fading when a vehicle is in motion can cause changes in signal strength of 20 dB or more within a few feet. The solution was to record signal strength from a continuously keyed base station over a distance while the test vehicle is in motion. The use of strip charts with an analog receiver in the test vehicle was an early implementation of this method. The value assigned to each test location was determined by either using the mean signal level (difficult to calculate from a strip chart) or the median signal level, which was straightforward and did not require calculations. The test location associated with the data was manually marked on a map at the time of the test.
With the advent of compact laptop computers and the deployment of the Global Positioning System (GPS) network of satellites in Earth orbit, the test equipment and data collection methods became more sophisticated. Test location could be determined with the use of a mobile GPS receiver, and recorded with the data file in the mobile laptop computer. The collection of signal strength data changed from the use of a strip-chart recorder to discrete samples of signal strength.
Analogous to the strip-chart recording, however, engineers realized that a single measurement would not be representative due to the effects of multi-path fading. The solution was to rapidly sample signal strength over a reasonable distance (such as 40 wavelengths), and record several hundred discrete samples of the signal strength. While the laptop computer could store all of the discrete samples (if sufficient disk space was available), typically the samples would either be averaged or analyzed for the median value. Either the mean or median value would be recorded and stored in association with the longitude and latitude position in the data file.
Calibration of the test equipment and the mobile radio setup is used to establish baseline signal strength, from which appropriate loss factors can be applied to derive or extrapolate performance for a portable radio (rather than the test mobile unit), as well as simulating portable use inside buildings of varying loss characteristics (e.g., light, medium or heavy buildings). Rather than introducing attenuators into the antenna feed lines, these loss factors can be accommodated by simply post-processing the resultant data files (i.e., subtracting the loss factor from the measured signal strength value).
Over the past ten years, the major land mobile radio suppliers have developed proprietary methods and tools for acceptance testing of radio systems that they field. Coverage test procedures, equipment and processing software have been independently developed by each company and applied uniquely to their system and equipment. Some vendors perform drive tests with continuous recording of data over the drive route, while others divide the service area into test grids and then record the coverage data on a sampled basis as they enter each grid during the drive test.
With the advent of digital voice modulation in land mobile radio systems, the method for coverage testing has been modified to include bit error rate (BER) measurements. Some in the LMR industry maintain that signal strength measurements cannot be directly correlated with BER, and, therefore, signal strength is not universally accepted as a valid measurement technique for digital voice coverage. However, it should be clear that BER measurements are not direct tests of the understandability of voice calls or of audio quality, and in actual practice the relationship can vary depending on the protocols used to convert between voice and the digital signal.
There are fundamental problems with all of the coverage test methods described above. For the audio quality test (circuit merit or delivered audio quality), there is subjective judgment by the people listening to the voice calls—what one person considers acceptable under one set of conditions as a DAQ 3 or 4, another person, under another set of conditions may judge to be unintelligible and unacceptable. With both the signal strength and BER tests, there is an assumed experientially derived correlation between the parameter being measured and the audio quality of the voice delivered to the listener. In both of these tests, the performance of the radio receiver is not taken into account, since the correlation made a general assumption concerning “typical” receiver capabilities. In the BER tests, the bit pattern transmitted and used for comparison at the receiving test unit is raw, i.e. not processed data. This means that the BER measurements do not include the voice/data conversion protocols—i.e., the capabilities of the manufacturer's vocoder or forward error correcting code, which can either contribute errors or correct errors in the digital stream. In addition, manufacturers infer a voice quality (delivered audio quality or DAQ) from their BER measurements. And in both BER and signal level coverage tests, the only path generally tested is “talk-out”—i.e., from base-to-mobile—making the tests unidirectional. The “talk-back” (mobile-to-base) path is more difficult to set up for automated testing, and is therefore seldom done—the assumption is usually made that the two paths are reciprocal, equivalent and “balanced”, meaning that both directions of the call have the same coverage footprint over the service area. This assumption can be reasonable in some situations, for example in a system where tower-top pre-amplifiers are utilized to balance talk-out and talk-back paths, but is probably a poor assumption in others, for example in systems where tower-top amplifiers are not practical. Systems built for the VHF (136-174 MHz) and UHF (406-512 MHz) frequencies are generally balanced where high-power mobile units are used, but are not balanced in the more common portable-based systems. Tower-top amplifiers are not effective at these frequencies due to the presence of high ambient noise. For digital systems, there is no direct means of post-processing BER data to extrapolate to other conditions. BER tests are only valid for the talk-out path under the existing conditions when measured, and cannot be easily related to the coverage for talk-back communications.
Thus, current automated LMR voice system field testing relies on measuring technical characteristics of the system, either signal level, BER, or a combination of the two over a distance of 40 to 100 wavelengths. These characteristics do not necessarily correspond directly to the understandability of the radio communications since there are additional factors involved in human communications. A direct human understandability test on the other hand is very time consuming, labor intensive, and not replicable from person to person or from time to time since there is a significant subjective element that differs with the perception of the test subject.
Current systems and methods implementing the Perceptual Evaluation of Speech Quality (PESQ) algorithms are also not capable of checking for missing or blank audio in recorded audio files and filtering such data from further processing.
What is needed is an automated process that emulates human communications, but removes human subjectivity, such as is found in MOS-based tests, from the test process. The desired method should include both the talk-out (base-to-mobile) and talk-back (mobile-to-base) directions, since a useable communications path must include the capability for two-way conversations. Furthermore, the test method should include a measurement of the entire radio system's processing capability (e.g., analog-to-digital conversion, digital-to-analog conversion, voice compression, forward error correcting code, receiver sensitivity, etc.) in the measurement. The method should be flexible enough to simulate the conditions that a radio user would experience, such as making or receiving calls with a portable clipped on his belt while inside a building with a specified typical loss factor. The method should also include the concurrent measurement of received signal strength to facilitate the identification of local conditions (e.g., noise, interference) that may affect the communications channel. Yet another function that should be provided by such a system and method is to check for and filter out missing or blank audio data. And, finally, the test method should be insensitive to proprietary control data, vocoders and other vendor-specific protocols. Ideally, the method should be non-invasive to the communications system under test: it should not require integrated access to the radio system components, instead relying only on typical interfaces presented by all systems (e.g., audio, microphone, push-to-talk, antenna, etc.). By being non-invasive, the method would facilitate independent verification and validation by a third party, and not rely on participation by the system manufacturer or provider.